{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# CPSC 330 hw4\n",
    "\n",
    "Note: this assignment covers 2 weeks' worth of material (lectures 6-9).\n",
    "\n",
    "Note: the assignments will get gradually more open-ended as we progress through the course. In many cases there is no single correct solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "from sklearn.dummy import DummyClassifier\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import Pipeline\n",
    "from sklearn.compose import ColumnTransformer\n",
    "\n",
    "# Add more imports below as needed\n",
    "\n",
    ""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Instructions\n",
    "rubric={points:5}\n",
    "\n",
    "Follow the [homework submission instructions](https://github.com/UBC-CS/cpsc330/blob/master/docs/homework_instructions.md). \n",
    "\n",
    "In particular, please make sure all your output is showing up in the final rendered version. **Also, please do not delete the question cells or move the questions around.** This will make things easier for the TAs.\n",
    "\n",
    "Also: if available, you are welcome to use scikit-learn functions for any of the tasks below, such as confusion matrix. You are not required to implement them yourselves. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Writing quality/quantity\n",
    "rubric={points:5}\n",
    "\n",
    "The TAs have reported a couple issues with the first few assignments: in some cases, submissions simply show the code output with no commentary; please write at least a sentence explaining your output in each question. In other cases, the TAs have come across multi-paragraph answers where a couple of sentences would have sufficed. Thus, we are now allocating the above points for well-structured answers of a reasonable length. In general, 1-3 sentences is good."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Customer churn data\n",
    "\n",
    "[Customer churn](https://en.wikipedia.org/wiki/Customer_attrition) refers to the notion of customers leaving a subscription service like Netflix. In this exercise, we will try to predict customer churn in a dataset where most of the customers stay with the service and a small minority cancel their subscription. To start, please download the [Kaggle telecom customer churn dataset](https://www.kaggle.com/becksddf/churn-in-telecoms-dataset). As usual, do not push the CSV to your repo. One you have the data, you should be able to run the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('bigml_59c28831336c6604c800002a.csv', encoding='latin-1')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train, df_test = train_test_split(df, test_size=0.1, random_state=123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>state</th>\n",
       "      <th>account length</th>\n",
       "      <th>area code</th>\n",
       "      <th>phone number</th>\n",
       "      <th>international plan</th>\n",
       "      <th>voice mail plan</th>\n",
       "      <th>number vmail messages</th>\n",
       "      <th>total day minutes</th>\n",
       "      <th>total day calls</th>\n",
       "      <th>total day charge</th>\n",
       "      <th>...</th>\n",
       "      <th>total eve calls</th>\n",
       "      <th>total eve charge</th>\n",
       "      <th>total night minutes</th>\n",
       "      <th>total night calls</th>\n",
       "      <th>total night charge</th>\n",
       "      <th>total intl minutes</th>\n",
       "      <th>total intl calls</th>\n",
       "      <th>total intl charge</th>\n",
       "      <th>customer service calls</th>\n",
       "      <th>churn</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>380</th>\n",
       "      <td>SD</td>\n",
       "      <td>137</td>\n",
       "      <td>510</td>\n",
       "      <td>373-5732</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>242.1</td>\n",
       "      <td>118</td>\n",
       "      <td>41.16</td>\n",
       "      <td>...</td>\n",
       "      <td>93</td>\n",
       "      <td>16.24</td>\n",
       "      <td>218.6</td>\n",
       "      <td>50</td>\n",
       "      <td>9.84</td>\n",
       "      <td>14.7</td>\n",
       "      <td>2</td>\n",
       "      <td>3.97</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2352</th>\n",
       "      <td>VA</td>\n",
       "      <td>118</td>\n",
       "      <td>408</td>\n",
       "      <td>404-2877</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>154.8</td>\n",
       "      <td>71</td>\n",
       "      <td>26.32</td>\n",
       "      <td>...</td>\n",
       "      <td>73</td>\n",
       "      <td>20.74</td>\n",
       "      <td>159.6</td>\n",
       "      <td>81</td>\n",
       "      <td>7.18</td>\n",
       "      <td>12.8</td>\n",
       "      <td>4</td>\n",
       "      <td>3.46</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>693</th>\n",
       "      <td>NJ</td>\n",
       "      <td>92</td>\n",
       "      <td>510</td>\n",
       "      <td>420-8242</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>29</td>\n",
       "      <td>155.4</td>\n",
       "      <td>110</td>\n",
       "      <td>26.42</td>\n",
       "      <td>...</td>\n",
       "      <td>104</td>\n",
       "      <td>16.02</td>\n",
       "      <td>254.9</td>\n",
       "      <td>118</td>\n",
       "      <td>11.47</td>\n",
       "      <td>8.0</td>\n",
       "      <td>4</td>\n",
       "      <td>2.16</td>\n",
       "      <td>3</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>527</th>\n",
       "      <td>NJ</td>\n",
       "      <td>95</td>\n",
       "      <td>415</td>\n",
       "      <td>379-6652</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "      <td>22</td>\n",
       "      <td>40.9</td>\n",
       "      <td>126</td>\n",
       "      <td>6.95</td>\n",
       "      <td>...</td>\n",
       "      <td>90</td>\n",
       "      <td>11.34</td>\n",
       "      <td>264.2</td>\n",
       "      <td>91</td>\n",
       "      <td>11.89</td>\n",
       "      <td>11.9</td>\n",
       "      <td>7</td>\n",
       "      <td>3.21</td>\n",
       "      <td>0</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2556</th>\n",
       "      <td>WA</td>\n",
       "      <td>118</td>\n",
       "      <td>510</td>\n",
       "      <td>422-2571</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>113.0</td>\n",
       "      <td>80</td>\n",
       "      <td>19.21</td>\n",
       "      <td>...</td>\n",
       "      <td>87</td>\n",
       "      <td>12.76</td>\n",
       "      <td>204.3</td>\n",
       "      <td>115</td>\n",
       "      <td>9.19</td>\n",
       "      <td>10.8</td>\n",
       "      <td>4</td>\n",
       "      <td>2.92</td>\n",
       "      <td>2</td>\n",
       "      <td>False</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 21 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "     state  account length  area code phone number international plan  \\\n",
       "380     SD             137        510     373-5732                 no   \n",
       "2352    VA             118        408     404-2877                 no   \n",
       "693     NJ              92        510     420-8242                 no   \n",
       "527     NJ              95        415     379-6652                 no   \n",
       "2556    WA             118        510     422-2571                 no   \n",
       "\n",
       "     voice mail plan  number vmail messages  total day minutes  \\\n",
       "380               no                      0              242.1   \n",
       "2352              no                      0              154.8   \n",
       "693              yes                     29              155.4   \n",
       "527              yes                     22               40.9   \n",
       "2556              no                      0              113.0   \n",
       "\n",
       "      total day calls  total day charge  ...  total eve calls  \\\n",
       "380               118             41.16  ...               93   \n",
       "2352               71             26.32  ...               73   \n",
       "693               110             26.42  ...              104   \n",
       "527               126              6.95  ...               90   \n",
       "2556               80             19.21  ...               87   \n",
       "\n",
       "      total eve charge  total night minutes  total night calls  \\\n",
       "380              16.24                218.6                 50   \n",
       "2352             20.74                159.6                 81   \n",
       "693              16.02                254.9                118   \n",
       "527              11.34                264.2                 91   \n",
       "2556             12.76                204.3                115   \n",
       "\n",
       "      total night charge  total intl minutes  total intl calls  \\\n",
       "380                 9.84                14.7                 2   \n",
       "2352                7.18                12.8                 4   \n",
       "693                11.47                 8.0                 4   \n",
       "527                11.89                11.9                 7   \n",
       "2556                9.19                10.8                 4   \n",
       "\n",
       "      total intl charge  customer service calls  churn  \n",
       "380                3.97                       3  False  \n",
       "2352               3.46                       0  False  \n",
       "693                2.16                       3  False  \n",
       "527                3.21                       0  False  \n",
       "2556               2.92                       2  False  \n",
       "\n",
       "[5 rows x 21 columns]"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The last column (`churn`) is the target. \"True\" means the customer left the subscription (churned) and \"False\" means they stayed.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1(a)\n",
    "rubric={points:4} \n",
    "\n",
    "Perform some exploratory data analysis on the training set. In particular:\n",
    "\n",
    "- How many rows and columns are there?\n",
    "- How many True/False target values are there?\n",
    "\n",
    "Come up with **two** more exploratory questions you would like to answer (similar to the above two), and explore those as well. Briefly discuss your results in 1-3 sentences.\n",
    "\n",
    "You are welcome to use `pandas_profiling` (see Lecture 6) but you don't have to."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1(b)\n",
    "rubric={points:10}\n",
    "\n",
    "In preparation for building a classifier, set up a `ColumnTransformer` that performs whatever feature transformations you deem sensible. This can include dropping features if you think they are not helpful. Remember that by default `ColumnTransformer` will drop any columns that aren't accounted for when it's created.\n",
    "\n",
    "In each case, briefly explain your rationale with 1-2 sentences. You do not need an explanation for every feature, but for every group of features that are being transformed the same way. For example, \"I am doing transformation X to the following categorical features: `a`, `b`, `c` because of reason Y,\" etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df_train.drop(columns=[\"churn\"])\n",
    "X_test  = df_test.drop(columns=[\"churn\"])\n",
    "\n",
    "y_train = df_train[\"churn\"]\n",
    "y_test  = df_test[\"churn\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1(d)\n",
    "rubric={points:3}\n",
    "\n",
    "The original dataset had a feature called `area code`. Let's assume we encoded this feature with one-hot encoding.\n",
    "\n",
    "1. The area codes were numbers to begin with. Why do we want to use one-hot encoding on this feature?\n",
    "2. What were the possible values of `area code`? \n",
    "3. What new binary feature(s) were created to replace `area code`? \n",
    "4. For each possible value of `area code`, how is this value represented in the transformed data? For example, for a different feature called `international plan` has two values, \"no\" and \"yes\". In the transformed data, there is new feature called `international plan_yes`, where \"yes\" is represented as 1.0 and \"no\" is represented as 0.0. (There may also be a new feature called `international plan_no` depending on what hyperparameters you used in your `OneHotEncoder` - either way is fine.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1(e)\n",
    "rubric={points:4}\n",
    "\n",
    "Create a `DummyClassifier` using `strategy='prior'`. Report the following scoring metrics via cross-validation: accuracy, precision, recall, F1-score. Briefly comment on your results, including any warnings the code produces (2 sentences max)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1(f)\n",
    "rubric={points:10} \n",
    "\n",
    "- Train and score a logistic regression classifier on the dataset. \n",
    "- Report the same metrics as in the previous part.\n",
    "- Are you satisfied with the results? Use your `DummyClassifier` results as a reference point. Discuss in a few sentences. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1(g)\n",
    "rubric={points:5}\n",
    "\n",
    "Set the `class_weight` parameter of your logistic regression model to `'balanced'`. Report the same metrics as in the previous part. Do you prefer this model to the one in the previous part? Discuss your results in a few sentences."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2: Hyperparameter optimization"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2(a)\n",
    "rubric={points:5}\n",
    "\n",
    "Now let's tune the hyperparameters of our `LogisticRegression` using `GridSearchCV` to maximize cross-validation accuracy. Jointly optimize `C` (choose some reasonable values) and `class_weight` (`None` vs. `'balanced'`). What values of `C` and `class_weight` are chosen and what is the best cross-validation accuracy?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2(b)\n",
    "rubric={points:3}\n",
    "\n",
    "This time, do the same optimization but have `GridSearchCV` optimize F1-score instead of accuracy. Discuss any changes in your results.\n",
    "\n",
    "_Checkpoint_: It should be choosing `class_weight='balanced'` when you're optimizing F1-score. I am getting an F1-score around 0.48. You don't need to have exactly this, but if you're seeing something completely different there may be a problem."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2(c)\n",
    "rubric={points:5}\n",
    "\n",
    "This dataset is fairly small so the code should be fairly fast to run. This makes it a good time to use more folds in cross-validation. Using your best model from the previous part, compute the F1-score using 20-fold cross-validation and print out the sub-scores. How big would you say the \"error bars\" are on your average score? (The answer is somewhat subjective, just try to be reasonable.) Briefly discuss."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 3: test set"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(a)\n",
    "rubric={points:5}\n",
    "\n",
    "- Evaluate your final model's F1 score on the test data. \n",
    "- Is it within your \"error bars\" from the previous part? (If it isn't, that doesn't mean you need to redo the previous part!) \n",
    "- Do you think you overfit on the cross-validation sets?\n",
    "- Briefly discuss your results (1-2 sentences)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(b)\n",
    "rubric={points:2}\n",
    "\n",
    "Create a confusion matrix for your final model - one on the training set and one on the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(c)\n",
    "rubric={points:5}\n",
    "\n",
    "It is possible to read the precision and recall directly off these confusing matrices by normalizing them - see the `normalize` argument in the [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.plot_confusion_matrix.html). \n",
    "\n",
    "1. Compute your final model's precision on the test set.\n",
    "2. Normalize the confusion matrix such that you can read the precision off it, and show how you can do this.\n",
    "3. Compute your final model's recall on the test set.\n",
    "4. Normalize the confusion matrix such that you can read the recall off it, and show how you can do this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3(d)\n",
    "rubric={points:5}\n",
    "\n",
    "The function below plots histograms of the predicted probability, split by the **true class**, for a given model and dataset. This is similar to the animated plots from lecture. \n",
    "\n",
    "Call this function twice, once for each of two models:\n",
    "\n",
    "1. A pipeline with your chosen value of `C` and `class_weight='balanced'`\n",
    "2. A pipeline with your chosen value of `C` and `class_weight=None`\n",
    "\n",
    "Then, discuss your results: how do the two plots differ? Is `class_weight` changing things the way you expected? \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [],
   "source": [
    "def make_hists(model, X, y):\n",
    "\n",
    "    negative_examples = X[y == 0]\n",
    "    positive_examples = X[y == 1]\n",
    "\n",
    "    plt.hist(model.predict_proba(negative_examples)[:,1], alpha=0.5, bins=30, label=\"0\", density=True)\n",
    "    plt.hist(model.predict_proba(positive_examples)[:,1], alpha=0.5, bins=30, label=\"1\", density=True)\n",
    "    plt.legend(loc='upper right')\n",
    "\n",
    "    plt.xlabel(\"predicted probability\")\n",
    "    plt.ylabel(\"normalized counts\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 4: Precision and recall by hand"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Below is the confusion matrix of a machine learning system that predicts whether a cancer is malignant or not. Let's consider malignant to be the \"positive class\".\n",
    "\n",
    "|    Actual/Predicted      | Predicted Benign | Predicted Malignant |\n",
    "| :------------- | -----------------------: | -----------------------: |\n",
    "| **Actual Benign**       | 6 | 238 |\n",
    "| **Actual Malignant**       | 20 | 194 |"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4(a)\n",
    "rubric={points:2}\n",
    "\n",
    "Would you consider this an imbalanced dataset? Why or why not? Max 2 sentences.\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4(b)\n",
    "rubric={points:2}\n",
    "\n",
    "Based on the above confusion matrix, what is the recall? \n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4(c)\n",
    "rubric={points:5}\n",
    "\n",
    "Do you consider this to be a good classifier? What additional information might you need to answer this question? Briefly discuss in 1-3 sentences.\n",
    "\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 5: Very short answer questions\n",
    "rubric={points:10}\n",
    "\n",
    "Answer each of the following questions in **1-2 sentences**. Each one is worth 2 points.\n",
    "\n",
    "1. One can think of `predict` as thresholding the output of `predict_proba` at some threshold. What is one scoring metric we talked about which is independent of this threshold? Briefly explain.\n",
    "2. What is the difference between stratified cross-validation and regular cross-validation?\n",
    "3. What is an advantage of ensembling multiple models as opposed to just choosing one of them?\n",
    "4. What is an disadvantage of ensembling multiple models as opposed to just choosing one of them?\n",
    "5. By default, `StackingClassifier` uses `LogisticRegression` as its \"meta-model\". Explain the significance of the coefficients learned by this `LogisticRegression` model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    ""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Submission to Canvas\n",
    "\n",
    "**IF YOU ARE WORKING WITH A PARTNER PLEASE FORM THE GROUP ON CANVAS BEFORE SUBMITTING** - see instructions [here](https://github.com/UBC-CS/cpsc330/blob/master/docs/homework_instructions.md#partners).\n",
    "\n",
    "When you are ready to submit your assignment do the following:\n",
    "\n",
    "1. Run all cells in your notebook to make sure there are no errors by doing `Kernel -> Restart Kernel and Clear All Outputs` and then `Run -> Run All Cells`.\n",
    "2. Save your notebook.\n",
    "3. Convert your notebook to `.html` format using the `convert_notebook()` function below **or** by `File -> Export Notebook As... -> Export Notebook to HTML`.\n",
    "4. Run the code `submit()` below to go through an interactive submission process to Canvas.\n",
    ">For this step, you will need a Canvas *Access Token* token. If you haven't already got one, log-in to Canvas, click `Account` (top-left of the screen), then `Settings`, then scroll down until you see the `+ New Access Token` button. Click that button, give your token any name you like and set the expiry date to Dec 31, 2020. Then click `Generate token`. Save this token in a safe place on your computer as you'll need it for all assignments. Treat the token with as much care as you would an important password. \n",
    "\n",
    "Note: for those having trouble with the Jupyter widgets and the dropdowns: if you add the argument `no_widgets=True` to your `submit` call, it should let you do a text-based entry of your key and avoid the dropdowns altogether. If this doesn't work, you probably need to upgrade to the latest version of `canvasutils` with `pip install canvasutils -U` from your terminal with your environment activated.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from canvasutils.submit import submit, convert_notebook\n",
    "\n",
    "# Note: the canvasutils package should have been installed as part of your environment setup - \n",
    "# see https://github.com/UBC-CS/cpsc330/blob/master/docs/setup.md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# convert_notebook(\"hw4.ipynb\", \"html\")  # uncomment and run when you want to try convert your notebook to HTML (or you can convert manually from the File menu)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# submit(course_code=53561, token=False)  # uncomment and run when ready to submit "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  },
  "name": "_merged",
  "toc": {
   "colors": {
    "hover_highlight": "#DAA520",
    "navigate_num": "#000000",
    "navigate_text": "#333333",
    "running_highlight": "#FF0000",
    "selected_highlight": "#FFD700",
    "sidebar_border": "#EEEEEE",
    "wrapper_background": "#FFFFFF"
   },
   "moveMenuLeft": true,
   "nav_menu": {
    "height": "438px",
    "width": "252px"
   },
   "navigate_menu": true,
   "number_sections": true,
   "sideBar": true,
   "threshold": 4,
   "toc_cell": false,
   "toc_section_display": "block",
   "toc_window_display": false,
   "widenNotebook": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
